To get started quickly, the code for this tutorial can be found at the following Github repo (here).
Loading Data
SWDI API
The SWDI data covers a number of data series that use event detection algorithms on NEXRAD radar data. The main data series are as follows:
- ‘nx3tvs’ - (Point) NEXRAD Level-3 Tornado Vortex Signatures
- ‘nx3meso’ - (Point) NEXRAD Level-3 Mesocyclone Signatures
- ‘nx3hail’ - (Point) NEXRAD Level-3 Hail Signatures
- ‘nx3structure’ - (Point) NEXRAD Level-3 Storm Cell Structure Information
- ‘plsr’ - (Point) Preliminary Local Storm Reports
- ‘warn’ - (Polygon) Severe Thunderstorm, Tornado, Flash Flood and Special Marine warnings
We’ll use the SWDI API to extract all hail signatures between 2005 and 2015 in 30 day incremenets as the maximum number of days per API call is 31 days or 744 hours. The wrapper ‘swdi_pull’ will need three parameters:
- start date
- end date
- series
swdi_pull <- function(start_date,end_date,series){
##translate the string into a date, and range
start <- as.Date(start_date,"%Y-%m-%d")
range <- as.numeric(as.Date(end_date,"%Y-%m-%d")-as.Date(start_date,"%Y-%m-%d"))
##Placeholder for the result
raw <- data.frame()
##Loop through Day 0 through the full range of days
for(i in seq(0,range,30)){
##Load in parameters, hit API
print(i)
period <- start + i
increment0 <- paste(format(period,"%Y"),format(period,"%m"),format(period,"%d"),sep="")
increment1 <- paste(format(period+30,"%Y"),format(period+30,"%m"),format(period+30,"%d"),sep="")
temp <- read.csv(paste("http://www.ncdc.noaa.gov/swdiws/csv/",series,"/",increment0,":",increment1,sep=""))
##If the API kicks back a result
if(ncol(temp)!=1 && colnames(temp)[1]!="summary"){
raw <- rbind(raw,temp)
raw <- raw[!is.na(raw$LAT),]
}
}
##Clean up time steps -- remove data outside of specified period
raw$DATE<-as.Date(substr(raw$ZTIME,1,10),"%Y-%m-%d")
raw<-raw[raw$DATE<=as.Date(end_date,"%Y-%m-%d"),]
raw$HOUR<-substr(raw$ZTIME,12,13)
raw<-raw[,c("ZTIME","DATE","HOUR","WSR_ID","CELL_ID","PROB","SEVPROB","MAXSIZE","LAT","LON")]
return(raw)
}
Now, we can tap the API for data. We’ll first specify the parameters.
##Set Parameters
start_date = "2005-01-01"
end_date = "2014-12-31"
range <- as.Date(end_date,"%Y-%m-%d")-as.Date(start_date,"%Y-%m-%d")
series = "nx3hail"
fraction = 0.25
And using those parameters, we can now draw down the data. This will take a while – about 10 million records.
raw <- swdi_pull(start_date,end_date,series)
Geographic Files
As the SWDI data is point-level data that will be processed into equal-interval grid points, we will want to add spatial context to the data by spatially joining points to county boundary files. The US Census Bureau provides boundary shapefiles through their website (http://www2.census.gov/geo/tiger/GENZ2014/shp/cb_2014_us_county_20m.zip). To efficiently load it in, we’ll write a simple function to download, unzip and load a shapefile.
shape_direct <- function(url, shp) {
temp = tempfile()
download.file(url, temp) ##download the URL taret to the temp file
unzip(temp,exdir=getwd()) ##unzip that file
return(readOGR(shp,shp))
}
To run the shape_direct function, we just need the url and the shapefile name.
shp <- shape_direct(url="http://www2.census.gov/geo/tiger/GENZ2014/shp/cb_2014_us_county_20m.zip",
shp= "cb_2014_us_county_20m")
## OGR data source with driver: ESRI Shapefile
## Source: "cb_2014_us_county_20m", layer: "cb_2014_us_county_20m"
## with 3220 features
## It has 9 fields
In addition, we’re going to pull in a reference table that links Federal Information Processing System (FIPS) codes that contain numeric identifiers for states. This’ll be useful for clearly indicating in plain language which counties are in a given state.
fips <- read.delim("http://www2.census.gov/geo/docs/reference/state.txt",sep="|")
fips <- fips[,c("STATE","STUSAB")] ##Keep the FIPS code and state abbreviation
fips$STATE <- as.numeric(fips$STATE) ##Convert FIPS code to numeric
fips$STUSAB <- as.character(fips$STUSAB) ##Convert state name to character
Basic Processing
At this point, we have all the data (hail signatures, county shapefiles, FIPs codes).
paste("Number of records in Hail data: ",nrow(raw),sep="")
## [1] "Number of records in Hail data: 9285845"
paste("Number of counties in shapefile: ",nrow(as.data.frame(shp)))
## [1] "Number of counties in shapefile: 3220"
Now, we can begin the process down the data. The first issue is to convert hail signatures from events to regularly spaced grid points at the daily level. As two or more radar stations may detect the same hail event at the same time, this basic gridding process will help to reduce double counting as well as allow for calculating a climatology.
To start, we’ll set a bounding box of the continental US in order to focus on non-island effects.
#Cut down bounding box
raw <- raw[raw$LON<(-50) & raw$LON>(-140) & raw$LAT > 25,]
Also, the hail event coordinates will be rounded to the nearest fraction as specified in the starting parameters. In this case, the fraction is 1/4 of a degree or about 17.25 miles latitudinally. Using SQL, we will group hail events by date, lat, lon, and hail size. Then based on the hail size, we’ll produce dummy variables are produced for each 2+ inch and 3+ inch thresholds. For context, that a baseball is approximately 2.9 inches.
##Round coordinates
raw$LON <- round(raw$LON/fraction)*fraction
raw$LAT <- round(raw$LAT/fraction)*fraction
##De-duplicate by day, latitude and longitude
deduped_day <- sqldf("SELECT DATE, LON, LAT, MAXSIZE
FROM raw
GROUP BY DATE, LON, LAT, MAXSIZE")
## Loading required package: tcltk
##Dummy variable (and 3+ in)
deduped_day$lvl_3in<-0
deduped_day$lvl_3in[deduped_day$MAXSIZE>3]<-1
Based on the de-duplicated daily, gridded data, we’ll now group by once again by lat and lon coordinates. This time, we’ll count the number of records per gridpoint (cnt = any hail event) as well as sum the dummy variables for 3+ inch (cnt_3) events. These count variables are then normalized by the number of days specified in the API call (range).
##Daily gridded frequencies
singles <- sqldf("SELECT LON, LAT,COUNT(date) cnt, SUM(lvl_3in) cnt_3
FROM deduped_day
GROUP BY LON, LAT")
##Normalize
for(i in 3:ncol(singles)){
singles[,i]<-singles[,i]/as.numeric(range)
}
